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ABSTRACT 

This presentation, by an invited speaker at the 46th 
annual meeting of the National Association for Research in Science 
Teaching, describes simulation techniques used in the medical 
education program at the University of Illinois. Medical students 
interact with simulated patients and acquire problem-solving 
competencies for use in working with patients. Simulation techniques 
described include a paper and pencil format using latent image or 
opaque overlay techniques (with feedback systems, a computer format 
employing unconstrained natural language, a computer-managed robot 
(SIM I) , and live simulations using role playing. The question of 
validity of performance on a simulation is considered from the 
perspectives of content or face validity, or construct validity, and 
of concurrent validity. Evidence from data suggests that, when 
simulation techniques are properly exploited, these exercises hold 
considerable promise of being a powerful tool not only for 
instruction but also for research into the nature of problem solving* 
(PEE) 
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SIMULATION TE(]INIQLJE IN THE TEACHING AND TESTING 

OF 

PROBLEM-SOLVING SKILL?* 
Ciiristine McGuir-, M.A.^* 

■'rQblc:-:i-solving j-.; integral part of living, requisite in e.very 
f .Lf-I^l of endeavor, au c-r:* =ige and for every condition of man. Jn 
uhesc days of protest aoout cultural and sexual biases in education it 
lo irr:or tan c to note that trie doF.and to solve problems is no respecter 
o" po.rsc:Ts: its imperatives are the v.ame irrespective of the age, race, 
se:<3 ^'lass, creed or color of the problem-solver, Ainong those who face 
a .^iven problem only the resources — not the task — will differ. All 
3ius;t bring to bear tie same range of skills in the sequential process 
■That begins with th.e initial definition of the problem and ends only 
v;ir.fi its resolution or with catastrophe. 



=^^?resent'^d ac t'-xo. 46th /mnual Meeting of the National Association 
lor Research in Science Teaching, Detroit, Kichigan, March 27, 1973. 

'^^Professur of Educational Psychology > College of Education and 
Froiessor of Medical Education, Assistant Director and Chief, Research 
and Evaluation, Center for Educational Developittent , College of Medicine, 

i/v: i V e r s 1 1: y o 1 1 1 i n o 1 s , Cli ic ago , 1 1 lin o i s « 



■•^.i:.- ?.:::\v. r.r.bt be brought into play in this task induce 

H ^•^'.\\v::\v\- a- d urucessing skills, data interpreting skills, skill 
I ■. :.\ vari-**.,y oi typcis of resources includiiig expert advice, 

-.vi : rcl'Lr prLorities of data seekii\g and decision, decision- 
^•r\ills, ii\ n^anipi.ilating. the situation to alt|e.r it> skills 

''r. -^.onf-lvrir.u die effects of these manipulations and readjjisting d^- 
cisions or actionc- to respond ■j,o ihe. irodified situation, 

Sy imi taring a sequential decision process which entails a com- 
bination of sruch skills, sitrulation offers an exciting technology for 
both the? reaching and the testing of probleni-solving competencies, For 
tiliis rtascp- some mor-j or less exotic simulation techniques have long 
enjoyed a deservedly :" ivo^ad status as instructional adjuncts in the 
education of business e>:ecutives (business and mianagement games), miii- 
tai-y per^-cnnel (var ^anies and training exercises) , professional pilots 
(ihe Link Trainer), and even astronauts (space flight simulators), 

'JovGver, dfspitH the extensive literature on the instructional uses 
o!: simulation, this technology has been employed in only very limited 
areas of professional education and the examples set by the demon- 
strably effective institutionc?! modalities using it have raiely been 
followed in public and university education. Furthermore, in most 
elementary and secondary programs genuine simulation of problem-solving 
activities (as opposed to role-playing and other less structured ganes) 
is infrequently employed as an instructional strategy, and its poten- 
tialities for assesi^ment have not even been explored in most areas. 
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Viie Essenr.e of Simulation 

Perhaps such llnitod jxploitation is In part due to the vision of 
.-as tropomi cal co•-■^s c jn.j 'i red up by the word "siirulator . Yet > reduced 
to Its essence, siniuLation consists merely in placing an Infiividual in 
a rcalisric: setting \':iero he is v.onfronted by a problematic situation 
tha: requires his ac :ive participation in initiating and carrying 
through a sequence c' inqairies, decisions and actions. The situation 
ir.v.st he. cesigvi'itd so :hat each of thc?se activities triggers appropriate 
feedback which can utilized for :.ubsequent decisions about pending 
action, deci:-;ions vhlc". xcill :'n turn ciodify the problem In different 
ways depending on th3 un-.c'ie configuration of reactions and interven- 
tions each person macev . in this fashion a simulation can be evolved 
through many stages :inti,l it is terminated when the individual reaches 
an acceptable rerrolution or is faced by disruptive alternatives brought 
abcut by his own decisions and actions. 

Recently this essence has been captured in various sldtilatlon m.o- 
dalities that are economically and technologically feasible to use In 
both the instruction and assessment of three critical components of 
physician competence: skill in interpreting clinical and laboratory 
data, judgment in patient management, and skill In dealing effectively 
witii patients and colleagues. Because I believe that analogous com- 
ponents of competenca are inherent in all problem-solving I* should like 
to share with you our experiences at the University of Illinois where 
medical students are learning, by interacting with simulated patients, 
how to cope with the actual daily problems that they will confront as 
physic i an^ . 



i 
t 

Alt ernative Simulation Techniques 
Interpret ive Skills 

la Che teaching and testing of interpretive skills it is clear 
th.'-it provision ?vjst be made for the fact that data in any field come 
".c the problem solver through many sensory modalities: visual data, 
auditory data, tactile data to name but a few. Consequently, in 

(.'ealing vith these i*nportant skills in physician education, simulated 

I 

clinical aiul laboratory data are presented to the student in realistic 
form and he is simply asked to interpret them. These simulations may 
be based on photographic reproductions of x-rays, gross and .tnicrc- 
scopic specimens, lesions, ECG or EEG tracings and the like~-all pre- 
L^e.nted in a form that imit:-tr^s life as closely as possible. Alter- ' 
natively, they may employ sound simulators that have been developed for 
playback of high fidelity recordings of heart, breath and abdominal 
sounds through individual stethophones so that the sounds come to the 
student physician's ears in exactly the samie manner that they would 
roach hir: iri real life. Data that involve a combination of sound, 
color and movement are presented in high quality movies or videotapes 
depicting the physician-patient interview and/or certain aspects of 
the physical examination. 

More recently several three-dim.ensional models (for example, heads 
with eyes, ears anc. throats) have been developed into whicA varied 
pathology can be inserted. V/ith these models the student physician 
can learn and can demonstrate that he has mastered the skil^ of using 
sophisticated instruments both to collect data and to interpret them- 
Perhaps the most elegant of the three-dimensional models now in 




existence, is that developed by the Univerr?ity of Southern California— 
SIM I, a computer managed, extraordinarily life-like robot who can be 
programmed to present combinations of findings that can be modified 
In an almost infinite number of ways for the edification and evalua- 
tion of anesthesiologists. 

Figure 1 
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Photographic Reproductions 
Sound Simulators 
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CLINICAL /CiT) ^ — Sound Movies and Videotapes 
LABORf\TOR-: DATA \>^^ *i 

Three-dim.ensional Models 

\ 

Automated Robot (SIM I) 



Using any or all of these modalities, the student can learn and 
can he required to demonstrate that he has learned to describe pre- 
cisely the findings presented in auditory, visual or other forms, to 
interpret; the significance of these findings, to make some initial 
judgments about their significance and interrelations, to anticipate 
other findings that might be related to those demonstrated, and to 
predict what might exacerbate or relieve the situation. The principle 
is simple: Do not use words to present data that normally present them- 
selves via other ser.sory modalities. With but a moment's thought the 
applications to any field are obvious. 
Problem-Solving Skills 

But interpretive skills constitute only one element in effective 
problen~9olvixig . T'^'-.us in the teaching and testing of physicians it is 
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recetiJr;ar> r.o incorporate clinical and lahcratcry data of the type des- 
cribed abc'/e ivto varxoas problem foraats that similate the total situ- 
ation confronting th3 physician working up and nianaging a patient. 
Ally exercis.v which purports to simulate this type of decision- 



FlTPt,. it 'zust be initiated in a realistic manner; that" is, 
the problem must be brought to the attention of the student 
in a form that resa7iible"s the form in which it w^ould be posed 
in the real world; it must not be introduced by a careful 
predigested sumtnary of the salient features of the situation. 

Second, the exercise must require a series of sequential, in- 
terdependent d'scisions representing the various stages in the 
definition J analysis, and resolution of tlie problem. 

Third, the student must be able to obtain information in 
realistic form about the results of each inquiry or action, 
as a basis for subceqc^nt action. 

Fourth, once these data are obtained it must be impossible 
for him to retract a decis^ion that was revealed to be inef- 
fectual or harTiful. 

Fifth, the problem must be constructed so as to allow for 
different approaches and for variation in feedback appropri- 
aj-:* to tliese several approaches. Hence, provision must be 
tnadj? for modifications in the Tsroblem In response to specific 
actions taken by each student* 

Finally, these modifications must differ among students accord- 
ing to the unique configuration of prior decisions each has 
made . 

Problems meeting these specifications have now been dev .xoped 
in four modalities lor use with medical students, residents and clini- 
cians: a paper-and-pencil format using latent image or opaque over- 
lay techniques for feedback systems, in problems which are suitable 
for ;2iither individu-il or group administration and ar^^^. amenable to com- 
puter scoring and analysis (1,2,3); a computer format employing un- 
ronsti^ained aa'iural language and suitable for individualized adminis- 
tration to untrained subjects (4) ; a com.puter-managed robot simulation 
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for individual instruction and assessment; and an oral interaction 
format suitable for role-playing exercises in individual or small 
group settings (356,7). 

In the written format — widely known as Patient Management Problems 
(PMP) — the clinical situation is introduced with a brief statement of 
a complaint such as a physician might elicit from the patient himself, 
from a friend or relative who accompanied him or from a colleague who 
referred him. Information about the patient's condition is couched in 
realistic terms appropriate to its source. It may consist of only a 
few lines or a brief film clip in which the patient indicates his rea- 
sons for "seeing the doctor." 

Figure 2 

ASSESSMENT OF PROBLEM-SOLVING SKILLS 



Written Simulations (PMP) 

SIMULATED Computer Assisted Simulations (CASE) 

CLINICAL 

PROBLEMS \ "Automated Robot Simulations 




Live Simulations 



The statement of the problem is followed by a list of specific 
interve-.ntions or of general strategies; from these the student se- 
lects his initial approach, e.g., further history, physical examina- 
tion, laboratory eviiluation or immediate therapy. He records his 
decision by erasing the opaque overlay or developing the latent image 
on a specially treat:ed answer sheet to reveal instructions directing 
him to the section of his choice. In each new section possible in- 
terventions that may yield further information about the patient are 
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lifted. The student .7'-:lects as many or as few as seem appropriate 
in light of the speci^'lc circumstances obtaining at that stage of the 
•problem. Once again, as he records each decision, the results of 
that intcr'/ention are r€'ver!led in realistic vetbal or visual form. 

Both the stages in the work-up and the responses to each inter- 
vention are desij^ned tc? simulate an actuaJ. clinical situation in the 
office or hospital setting. For ex.ample, an order for a specific test 
reveals s. laboratory report; an order for an x-ray, electroencephalo- 
gram, electrocardiogram, audiogram, or the like directs the physician 
to a photographic reproduction of the x-ray or tracing; a color plate 
is provided in response -c an order for a smear, culture or biopsy; 
the patient's reaction ^ r-^^^orted in response to medication. Unless 
a consultation is requested, no interpretation of these data is offered 
c.nd none is explicitly deinande'^; as in a conventional clinical setting, 
data are ordinarily provided in response to any request and these data 
are available for consideration in making subsequent decisions about 
the management of the patient. 

In this manner a problem is carried through many stages, in each 
of which the student, must make further choices based on the specific 
response!? of the patient which were evoked by his own earlier decisions. 
The com.plications which he must manage will depend, as they do in the 
office or clinic, on the unique combination of specific procedures he 
elected ,it earlier rtages of the problem. If his approach ^ntails 
harmful Interventions at any stage or fails to encompass measures es-- 
sential to the recovery of the patient, feedback is given regarding 
the complication that has developed and opportunity is provided to 
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tak- appropriate laeasures to rectify the patient condition » If 
these remedial measures are inadequate, the problem may bo. termina- 
ted and the student advised that the patient has suffered a relapse 
and has been sent to another hospital, or has been referred to a 
consultant, or has died. Alternatively, if the corrective measures 
are adequate > the student may be directed to a nev section of the 
problem representing the next stage in the work-up and management of 
that patient. Each simulation is thus composed of many sections which 
correspond to the various stages in the diagnostic work-up and treat- 
ment of the patient described, and at each stage alternate sections 
may be itinluded to provider for differing approaches that physicians 
may wish to take. 

The essence of this techr.ique is that it tries to imitate in 
paper and pencil form the kind of decision-m.aking process which the 
physician must go through in a real life situation. It imitates that 
process in several important respects: First, the student has con- 
siderable latitude in choosing his own approach to any patient problem 
presented. Second, he receives prompt feedback on the results of any 
inquiry or intervention he makes and this feedback is available to 
him for subsequent decisions about the work~up and management of the 
patient. Finally, as in real life, the complications that he will need 
to handle will vary depending on his own unique pattern of decisions. 

This type of paper and pencil simulation has been found suitable 
for the development of problems that vary not only with respect to 
content but with respect to the nature of the situation. For example, 
it has been utilized for problems ranging from emergency to long-term 
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chrur/lc uiyeeise situations; i:or problems that focus primarily on rnan- 
ager;..ent and involve special considerations in ironitoring the patient's 
procuress and revising frherapy accordingly, and for problems that re- 
quire the utilii^ation of paramedical and cocmiunity resources as well 
as for those in which organic disease is the primary consideration. 

Recent technical developments enable us to produce the latent 
image answer sheets for these simulations in small quantities > by means 
available to any classroom teacher and at a cost comparable to that of 
ordinary mimeographe'i materials. With this breakthrough, written simu- 
lation can become a reality in any instructional or testing program. 
We have, in fact, recently been working with several groups who have 
designed written simulation?" :»n problems in every discipline ranging 
from the authentication of an artifact found on a carefully described 
dig, through the selection of the best site and technique for catching 
trout in a particular stream to finding the least risky way to get a 
speciried shipment of contraband smuggled through customs. Surely every 
one can find a tempting morsel in that pot pourrl. 

A rationale analogous to that used in the construction of written 
sinnilations has been employed in the development of computer-assisted 
sim ulations of the clinical encounter — familiarly known as CASE. 
Tnough focusing more on diagnostic than on management skills, the com- 
pfiLer simulation as developed at the University of Illinois College 
of :.edicine has the advantage that the student can address the ^'com- 
puter patient" in unconstrained natural language without reference to 
a list of possible interventions that would either restrict or cue him 
unnaturally. 
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In utilizing CASE the student seats himself at a computer terminal 
where he is given a brief initial description of the patient's present- 
ing complaint. It is then his task to determine what data he needs in 
the patient work-up. He does so by interrogating the coir.puter directly 
just as he would question a live patient and the computer responds to 
his inquiries in natural language. Similarly, he can indicate each as- 
pect of the physical and laboratory examination that he would under- 
take with this patient. Again, the coirputer responds in appropriate 
natural language with a description of the physical finding elicited 
or w±th a laboratory report. On completion of his work-up, the student 
records his working dia£;r.osis and his plan of management. His entire 
decision-making process car. then be retrieved from the computer and ob- 
jectively scored and analyzed for feedback to him and his instructors. 

Third, the automated r obot? SIM referred to earlier, can be 
programmed not only to display various combinations of findings, but 
also to present a number of problems and to respond appropriately to 
different interventions of the anesthesiologist. Indeed, SIM I will 
do everything including vomit on the student who gags him when insert- 
ing an endotracheal tube. SIM may even '*die'^ if the student manages 
him inappropriately or responds inadequately to changes in his con- 
dition. However, the lovely thing about this patient is that he never 
gets tired of being examined or treated, and he can be raised from the 
dead as often as necessary for the education and assessment of anes- 
thesiologists. 

Realistic clinical problems can also be posed in the form of live 
sipmlations . In these simulations an actor, a housewife, another student 
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or aimos : «'inyone can bi* "piogranimsd^^ to simulate a patient in an inter- 
view setting. In a diagnostic interview for example, it is the stu- 
dent's task tic interview the programmed patient and elicit essential 
data fro'u him; in a management or therapeutic interview it is his task 
to cope *vith the tot,^l problem the patient may present. Skills in 
data gathering, in crisis management, triage, and office and patient 
managemeat have been explored by a variety of such live simulations. 

These simulatio-is differ from the usual games and role-playing 
exercises in that in programming the simulator nothing is left to chance. 
The authors of the sitnulation know exactly how he will respond to each 
gambit of the student, .:nd this is determined by the history and per- 
sonality of the patient -^i- '.\;?.ted. In ^ telephone simulation (8), for 
example, the student may be seated in an office-like setting when the 
telephone rings and a frantic voice on the other end of the line reports 
sc^e eirergency — an accident, a child suspected of ingesting poison, a 
suiciu- threat or the like. The student is required to respond as he 
vould tc a real-life emergency, the adequacy of his response can be 
reliablv judged and discussed with him, and he can be presented with 
replications of the same type of emergency until his responses indicate 
that he has mastered that type of problem. Similarly, in an office 
simulation the stude:nt may be confronted with a series of "patients" 
each of whom presen::s a very different problem to which he must respond 
in varied ways and :hese, too, can be replicated in fresh and interest- 
ing ways; until he has mastered each. 

!n short, wrltren simulations, computer simulations, automated 
roboLfci and live simulations can all ue used to present clinical prob- 
lems in a realistic, term that requires the student physician to respond 
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as he would in the office or clinic setting, that permits him to re- 
ceive life-like feedback from the simulator and that forces him to 
cope with the patiences responses to a sequence of inquiries and in- 
terventions, much as he will need to do in his actual practice. 
Interpersonal Skills 

Finally, the physician's interpersonal and communication skills 
can be assessed by simulated interviews and conferences in which an 
individual — again, not necessarily an actor — is programmed, to take the 
part of a patient, a colleague or other member of the health team. 
These simulations differ from the live simulations of clinical problems 
only in their focus; in the former, the emphasis is on solving a com- 
plex medical problem; it? t'^e latter, the situations are chosen to focus 
on the physician's communication skills, his sensitivity to patient and 
colleague needs and his competence in working with these important people 



For example, ir. what purports to be a straight-forward diagnostic 
interview, the simulated patient may, if appropriately handled by the 
student, provide cues that the real problem is a deep-seated personal, 
familial or emotional one. The student's sensitivity and response to 
t:hi'ae cues and his skill in handling such a patient, in communicating 
M'lih him and In gaining his understanding and cooperation in a proposed 



Figure 3 
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•plau of r?.nager.^.»_nt arc usually onlianced and arrt readily assayed by 
use of such simulatei iT\ter\^ie.ws . 

.'\naIogous iinvjlated interviews with colleagues have been developed 
r.o deal with referral and consultation requests and even with those 
ticklish situations in which colleagues differ sharply about the man- 
agement of a particular patient. Clearly, in such settings the stu- 
dent ^s skill and his professional attitudes are soon apparent both to 
\iini and to his instructor or examiner. 

Finally, the si-jiulated interview may be designed to require • the 
student to communicate with one or more members of the health team. 
For example, he may be required to give instructions to a simulated 
nurse or to request assi^ta-^cc for his patient from a simulated dieti- 
tian or social worker, or to make a presentation and to respond to the 
reactions of individuals who have been programmed to take the role of 
oti^er members of the health team in a simulated staff conference. Once 
again, the student can be rated on his communication skills, his inter- 
personal skills and his professional demeanor in that kind of situation. 

Advantages and Limitations of Simulation 
as an Educational Tool 
In examining the advantages and limitations of simulations of the 
type described above, one must ask about their advantages and limita- 
tions compared to what and for what purpose. In these discussions I 
have often been challenged with a sarcastic comment to the effect that 
if I think simulations of reality are so great for both teaching and 
testing why not use reality itself (i.e. actual patients). In answer, 
L must confess that for most purposes I regard simulation as superior 
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not only to conventional methodologies but also to reality. You see 
simulation imitates , it does not duplicate , life; this is both its 
greatest advantage a^d, in the minds of some, its greatest limitation. 

Clearly, there are some aspects of reality that cannot at the 
present time be economically simulated, if they can be simulated at all. 
Thus, it is necessary to be very cautious about predicting on the basis 
of responses to a srnulated situation how an individual will behave in 
reality; at best it is possible to predict how well that individual is 
capable of behaving.* Secondly, it is important to recognize that simu- 
lation is not an appropriate method for teaching or for testing all as- 
pects of performance. For example, simple recall of factual information 
is more economically an-^ dr'rectly measured by conventional techniques 
of objective testing. At the other extreme, professional habits can 
be assayed only by careful and repeated observation over a long period 
of time in diverse settings. Between these two extremes simulation 
providas the following important advantages. 
Perceived R.elevance 

As compared with more conventional instructional and evaluation 
strategies, simulation offers the possibility of designing problems 
so that they correspond more closely to the life situations a student 
actually faces. Learning materials, diagnostic tests and summative 
evaluations composec. of such problems appear to the student as far more 
relevant than typiceil textbooks or conventional multiple-choice exami- 



^ThcL same caution must, of course, be observed in predicting how 
nw individual will beliave in a new life situation, on the basis of how 
lu- behaved in a prior, necessarily somewhat different, life situation. 
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nations. Such perceived relevance is at the very least psychologically 
beneficial. And, in these days of concern about the cultural biases 
cf our schools uid the testing programs they promulgate, perceived rel- 
evance may turn out to be our only legal defense. 
Predeterm ination and Pre selection of the Task 

It is also of prime importance that by utilizing simulation tech- 
niques this perceived relevance can be achieved without being dependent 
on the accidents of nature and the flow of real problems available at 
the particular place and the specific moment in time when instruction 
is to be given or an assessment is to be made. It is obvious that simu- 
lation makes it possible to predetermine precisely the exact task which 
students are to be reauirc--^ to perform. Further, it is clear that in 
contrast with the ^'noise" always present in reality, simulation makes 
it possible to focus on the elements of primary' concern in a learning 
or a testing situation and to eliminate irrelevant and confusing com- 
plexit5.es that would complicate the learning and contaminate the assess- 
ment. In addition, the fact that learning and assessment tasks can be 
pre--selected means that they can be carefully graduated in difficulty 
as the student progresses through the educational program. Third, by 
developing standard parallel simulations it is possible to confront 
the student over and over again with interesting variations in what is 
essentially the same task until he has mastered it. 
Standardization of the Task 

Just as a given student can be repeatedly confronted with the same 
task, simulation enables an examining body to standardize the task for 
all examinees and, in the case of medical examinations, to do so without 
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subjecting one or a few patients to repeated bombardment by a large 
number of students. In short, all examinees can be given exactly the 
saiTie problem to cope with and this can be accomplished without an at- 
tack on nature. 

Improved Sampling of Performance 

3y standardizing the tasks and focusing on the most significant 
aspects in each it is possible in a given time period to instruct the 
student and to sample his performance vxth respect to a much broader and 
more representative group of problems. This, of course, requires care^ 
ful selection and deEinition of the tasks, but that is precisely the 
luxury simulation affords, which reality can rarely provide in a rea- 
sonable time frame. 
Improved Rating of Performance 

When the exact tasks that are to compose any learning or examina- 
tion experience are precisely defined and pre-selected it is possible 
to develop specific, detailed criteria for judging student performance, 
and to train teachers and examiners in applying these criteria equit- 
ably, consistently and in a manner to provide feedback of maximum bene- 
fit to the student. With this type of preparation of teachers and 
examiners there is no question that a higher degree of interrater re- 
liability can be achieved in the scoring of examinee performance with 
simulated problems and simulated interviews than with real problems. 

In our own studies correlations between repeated scorings of written 
simulations by different experts range in the neighborhood of .95 - ,98, 
That is as good as you can get in conventional multiple-choice testing 
where you normally have to rely on such dessicated slices of life that 
both the student and the teacher are justifiably **tumed off," 
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Increased Responsibility and Realistic Feedback in a Practical 
Time Fra:?.e 

One of the most important advantages of simulation over reality 
consists in the fact that all students can be allowed full responsi-- 
bility to careen dovr:i their merry way to disaster without any risk 
whatsoever to anyone! This is important not only in training pilots 
but in every professional arena, not least of which is the career of 
life itself. For example, in medicine the student can be allowed to 
continue along his appalling course and to receive realistic feedback 
without harm to anyone- Subject to the not so tender mercies of the 
neophyte, a simulated patient can become progressively sicker, can de- 
velop iatrogenic complica'zions , can die, contmic suicide, and yet be 
repeatedly revived, tireTassly to serve the same- or other students. 

Furthermore, if you wish to provide the studer: with experience 
and to test his mastery in the handling of chronic disease it isn't 
necessary for him., or for you, to wait 50 years watching, the gradual 
deterioration of a real patient. In carefully developed simulations 
a j if etiT.e of chrcnr'c disease can be collapsed in^*^ half-hour prob- 
lem, at each stage cf which the student can be piovided with feedback 
about his intervention-:^ in a "orrr. whic}i is more instructive than life 
i t: s elf u sual 1 y y iel c s . 
Increased Learning . 

The nature of the feedback provided in simulations as compared 
vith that available in either conventional educational settings or in 
reality should be underscored. Wh.ile the former rarely provides any 
genuinely relevant feedback, the latter often furnishes ic only in- 
directly, equivocally and long delayed • In contrast, the prompt. 
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specific and unambiguous fesidback characteristic of well designed sim- 
ulations makes them a powerful tool for the enhancement of learning. 

It is a genuinely exhilarating experience to walk through the 
halls after a group of our students have been involved in any of the 
clinical simulations described above. You find them busily checking 
with each other on the survival rates of their '^patients admonish- 
ing each other on the pros and cons or. various approaches to each, and 
s;>7earing off certain practices for life. It is not at all uncommon to 
hear such remarks as, *'I'll never again prescribe without taking a full 
drug history; I lost Mrs ^ Jones in surgery and I deserved to because I 
hadn^t bothered to find out that she was on steroids when she came in." 
And, of course, we have the usual quota of true stories about the num- 
ber of students who, after every examination, call the school psychia- 
trist in a state of severe anxiety because they've "killed a patient.'* 
lie may be only a paper patient but there no denying the students are 
mightily involved with him, and most of us would have to agree that 
students learn more, more deeply and more joyfully when they are so 
involved. 

Validity of Simulations 
This brings us to an exceedingly important question: How valid 
are the inferences made about performance on a simulation? In answer 
to this question, we have looked at three types of evidence: evidence 
about the nature of the intellectual process which the student must 
go through in order to respond to a simulation (content or face va- 
lidity); evidence about the extent to which the performance of dif- 
ferent groups on these exercises corresponds with reasonable hypotheses 
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about the way in which such groups should differ (construct validity); 
and evidence about the relation between scores on such exercises and 
performance on other tests or in other settings (concurrent validity). 
VJe hope later to have some evidence about the extent to which per- 
fomjance on simulated problems in patient management accurately pre- 
dicts some relevant aspect of future performance as a physician (pre- 
dictive validity), but that is not yet available. 
Content or Face Validity 

Repeated studies of the intellectual process required in respond- 
ing to Patient Management Problems and other simulations have been 
conducted in which experts who review the problems and students, resi-~ 
dents or physicians in practice who take them, have been asked both to 
analyze the exercise and to introspect about the cognitive process re- 
quired in responding to it. These studies can be summarized as follows 
Though no one claims that a simulation duplicates the clinical encoun- 
ter, there is clear agreement from all studies that the process of re- 
sponding closely approximates the process of thinking which one must 
go through in making decisions about the work-up and management of 
patients who present themselves in the office, clinic or hospital set- 
ting. ^Hiile criticisms have been leveled at specific problems (e.g., 
that the setting was not adequately specified, or that the physician's 
role was not adequat:ely Identified, or that options which the indi- 
vidual student wishe'.d to have included were not always to be found) no 
general criticism of the PMP format has been turned up in the system- 
atic studies that hnar on the issue of content or face validity. 
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Cu n s t m et: V al icl i. ty 

Studies of the extent to which differences in performance among 
different groups are compatible with reasonable hypotheses about what 
these differences should be have yielded exceedingly interesting re- 
sults; First, scores on the diagnostic portion of Patient Management 
Problems do not improve overall with more education and experience; 
in fact, they actually decline with age. Repeatedly, we find that 
groups with more education and experience request progressively less 
diagnostic information — even information thought to be essential by a 
criterion group — before making a therapeutic decision. Fortunately 
or not, this finding on simulations corresponds with findings from 
observational studies of sr'milar groups. 

On the other hand, increasing amounts of education are associated 
with somewhat higher scores on the therapy sections of these problems. 
This relation between education and performance is strongest in those 
questions which require prompt action in the absence of complete data 
(as, for example, in accident cases) and in those questions which re- 
quire radical action (as, for example, amputation). In short, we find 
over and over that the most experienced physician is most likely to 
take diagnostic short cuts and is most willing to take decisive action 
in treatment. And, not surprisingly, at all age levels this pattern 
of behavior is most characteristic of tho&e physicians who are psycho- 
logically most remote from academic settings, i.e., those who are ex- 
clutiively in individual private practice. 

In summary, differences in the responses of various groups to 
simulations we have studied are in the expected direction and the 




patterns of responses are closely similar to those reported in obser- 
vational studies of physician performance in clinical settings. 
Concurrent Validity 

Studies of the performance of medical students, residents, and 
board candidates for certification are uniform in revealing rather low 
correlations between scores on simulations and scores on multiple-choice 
tests, even those purporting to test ability to interpret data and to 
solve problems. In our experience the correlations range from approxi- 
mately .20 to approximately .40 even when both sets of exercises are 
quite lengthy. This is as it should be if different aspects u£ compe- 
tence are sampled by the *-vn types of exercises. Higher correlations 
would be very suspicious ir view of the nature of multiple-choice tests 
as they are currently employed. 

Unfortunately 5 stu<^.ies of the relation between performance on the 
si-nulations and supervisors' ratings of clinical competence have also 
Yielded very low correlations- This would be most discouraging were 
it not for two Important limitations of the ratings themselves. First, 
the ratings are not pure ratings of clinical judgment or problem- 
solving; they arc inescapably contaminated by the supervisors' impres- 
sions of the man's knowledge, skills and attitudes. Second, the re- 
liability of the ratings themselves is highly questionable; correla- 
tions between different supervisors' ratings of the same individuals 
on the same factors was generally not in excess of .30. This neces- 
sarily means that correlations between the supervisors' ratings and 
test performance are jeopardized by the low reliability of the ratings 
themselves. To date we have found no solution to this dilemma . 
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Studies of the relation between performance on simulations and 
actual clinical performance would seem to provide the best evidence 
of the concurrent validity of this format ^ when actual performance can 
be objectively auditad. But this is a very tedious and hazardous busi- 
ness. Our own early studies, admittedly limitec^ , ch/^T.TO(i sufficient 
congruence between these two measures to constitute exceedingly strong 

1 

presumptive evidence of the validity of well constructed simulations 

i 

as a measure of '^real-lif e'' performance. However, more receAt data are 
causing us to re-examine our comfortable assumptions about the nature 
and generalizability of problem^solving skills. And that brings me to 
my final and perhaps most important point, the nature of the problem- 
solving process itself. 

The Nature of Problem-Solving — A Question 
In any group, scores on any single simulation ordinarily distribute 
ther selves in the manner shown in Figure 4: Some students will be both 
chorough and discriirinating; others will take a very constricted ap- 
proach; still others; will take a shot-guB approach and some, hopefully 
few, will do a lot more harm than good. 

However, in our experience the correlations between scores on 
different problems may be very low. In other words, a student's suc- 
cess in dealing with a cardiovascular problem may be only miriiinally 
related to his succe-.ss in dealing with a renal problem; or his success 
in dealing with a chronic disease problem may not be very closely re- 
lated to his success in dealing with an emergency problem. You may 
say: "Of course, what else did you expect? No one does everything well*" 
If that is your re5;ponse I must assure you that I am not raising the 
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Figure A 

TYPICAL SCORES ON PMP SIMULATION 
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hoary issue of transfer of training across chasms. Let me remind you 
that I am not talking about the genius in theoretical physics who can- 
not change a clogged carburetor. Rather, I am talking about the stu- 
dent or the physician who, though he can recognize and manage backache 
attributable to disc disease, cannot recognize and manage backache 
attributable to claudication of the iliac artery, much less backache 
attributable to psychogenic factors! Now as a charter member of the 
Progressive Education Movement of the thirties, that kind of perfor- 
mance offends me« 

If I am correct in interpreting the frequent discussions of prob- 
lem-solving in your professional journals, you, too, seem to design 
your instructional and testing programs on two assumptions: first, that 
there are some general problem-solving strategies that are widely ap- 
propriate; and, second that experience or mastery in solving one prob- 
lem is correlated with and probably enhances problem-solving skills 
on a closely similar problem. If I am correct in believing that these 
two assumptions do, indeed, underlie your educational programs, then 
you, too, should be offended by the performance of my physicians. 

I have only three possible explanations of the intra- individual 
variability we are finding from problem to problem: first, the apparent 
content dependence of problem- solving skills may be only an artifact 
of variation in the formal structure of the various simulations; al- 
ternatively, it may be a consequence of the fragmented, disease oriented 
instruction characteristic of most medical schools; or finally, and most 
interesting of all, content dependent performance may not be a mirage, 
it may be a very real fact of life! I will let you know when I find 
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out, but I warn you aow that if it is a fact of life, you'd better 
know the answer to your problem before you select a physician to solve 
it. Otherwise my best advice is: Stay well! 

Summary Comment 

In summary, the evidence from studies of the use of simulation 
at several levels of medical education and with a variety of types of 
groups suggests that when these techniques are properly exploited, 
simulation exercises give cousiderable promise of being an extraordi- 
narily powerful tool not only for instruction and assessment of more 
relevant objectives, but also for research into the very nature of 
that most complex objective of all — problcm-solviag and judgment. 
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